System and method for label-free identification and classification of biological samples

ABSTRACT

A system and method of analyzing a biological sample using an imaging system are disclosed. An image acquisition module instructs the imaging system to obtain a label free image of a training biological sample and in response receives a first training image. The image acquisition module also instructs the imaging system to cause the training biological sample to fluoresce and obtain an image of the training biological sample undergoing fluorescence, and in response receives a second training image. An analysis module analyzing the second training image to generate a plurality of training cell characteristics, wherein each of the plurality training cell characteristics is associated with one of a plurality of training cells that comprise the training biological sample. A training module trains a machine learning system using the first training image and the plurality of training cell characteristics to develop a trained machine learning system such that when the trained machine learning system is operated with the first training image as an input, the trained machine learning system generates a plurality of predicted cell characteristics that correspond to the plurality of training cell characteristics.

FIELD OF DISCLOSURE

The present subject matter relates to a system and method for label-freeidentification and classification of biological samples and moreparticularly, to identification and classification of biological samplesusing a supervised machine learning system.

BACKGROUND

An imaging system such as a high content imaging system may be used tocapture one or more images of a biological sample. Thereafter, theimage(s) is/are analyzed to obtain metrics associated with thebiological sample. These metrics include, for example, a number ofdifferent types of cells in the biological sample, the number of cellsof each type, a proportion of living cells to dead cells, and the like.

The imaging system includes one or more illumination sources, one ormore objective lenses, and an image capture device such as acharge-coupled device or a complementary metal-oxide-semiconductor(CMOS) device to produce magnified images of the biological sample. Theillumination source may include a laser or other light source thatdirects light through the biological sample when the biological sampleis not undergoing fluorescence, and the light transmitted through thebiological sample is imaged by the image capture device. The image ofthe non-fluorescent biological sample captured by the image capturedevice is referred herein as a label free image and may be, for example,a transmitted light image, a phase contrast image, differentialinterference contrast image, and the like. As would be apparent to oneof ordinary skill in the art, these different types of label free imagesare transmitted light images captured under different illuminationconditions (e.g., with polarized light, light of particular wavelengths,etc.) and/or using different types of filters between the light sourceand the image capture device.

In some applications, the biological sample may be stained (or labeled)using one or more fluorescent dye(s), each of which adheres to one ormore particular protein(s) or component(s) in the biological sample. Theillumination source is used to expose the labeled biological sample tolight having particular characteristic(s) (e.g., a particularwavelength) that causes the fluorescent dye to fluoresce, and the lightemitted by such fluorescence is then captured by the image capturedevice. Compared to the transmitted or reflected images of unlabeledbiological samples, images of the fluorescence of the biological labeledwith the dye more clearly distinguish those elements (e.g., proteins andorganelles) associated with the fluorescence from other parts of thebiological sample. Thus, more accurate metrics regarding such elementsmay be developed from the image of the biological sample undergoingfluorescence.

However, labeling a biological sample with one or more fluorescent dyesis time consuming. Further, certain fluorescent dyes may be harmful, andeven toxic, to biological samples. For at least these reasons,developing images of labeled biological samples undergoing fluorescenceis not feasible or desirable in certain situations.

It would be useful to obtain the analysis advantages provided by imagingbiological samples under fluorescence using only transmitted lightimages.

SUMMARY

According to one aspect, a method of analyzing a biological sample usingan imaging system includes the step of causing a non-transitoryprogrammable device to undertake the steps of instructing the imagingsystem to obtain a label free image of a training biological sample andin response receiving a first training image, and instructing theimaging system to cause the training biological sample to fluoresce andobtain an image of the training biological sample undergoingfluorescence, and in response receiving a second training image. Thenon-transitory programmable device is caused to undertake the additionalsteps of analyzing the second training image to generate a plurality oftraining cell characteristics, and training a machine learning systemusing the first training image and the plurality of training cellcharacteristics to develop a trained machine learning system such thatwhen the trained machine learning system is operated with the firsttraining image as an input, the trained machine learning systemgenerates a plurality of predicted cell characteristics that correspondto the plurality of training cell characteristics. Each of the pluralitytraining cell characteristics is associated with one of a plurality oftraining cells that comprise the training biological sample.

According to another aspect, a system for analyzing a biological sampleusing an imaging system includes a machine learning system, an imageacquisition module, an analysis module, and a training module. The imageacquisition module operates on a non-transitory programmable device andinstructs the imaging system to generate a label free image of atraining biological sample and in response receives a first trainingimage, and instructs the imaging system to cause the training biologicalsample to fluoresce and obtain an image of the training biologicalsample undergoing fluorescence, and in response receives a secondtraining image. The analysis module operates on the non-transitoryprogrammable device and analyzes the second training image to generate aplurality of training cell characteristics, wherein each of theplurality training cell characteristics is associated with one of aplurality of training cells that comprise the training biologicalsample. The training module operates on the non-transitory programmabledevice and trains the machine learning system using the first trainingimage and the plurality of training cell characteristics to develop atrained machine learning system such that when the trained machinelearning system is operated with the first training image as an input,the trained machine learning system generates a plurality of predictedcell characteristics that correspond to the plurality of training cellcharacteristics.

Other aspects and advantages will become apparent upon consideration ofthe following detailed description and the attached drawings whereinlike numerals designate like structures throughout the specification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an imaging system;

FIG. 2 is a block diagram of a training and analysis system;

FIG. 3 is a flowchart of the steps undertaken by the training andanalysis system of FIG. 2 to train a machine learning system thereof;

FIG. 4 is a flowchart that shows in further detail the steps undertakenby the training analysis system of FIG. 2 to train the machine learningsystem thereof;

FIG. 5 is a flowchart that shows the steps undertaken by the trainingand analysis system of FIG. 2 to analyze an image of a biological sampleusing a machine learning system trained in accordance with theflowcharts shown in FIGS. 3 and 4;

FIG. 6 is an example of a process flow of training a machine learningsystem of the training and analysis system of FIG. 2; and

FIG. 7 is an example of a process flow of operating a trained machinelearning system of the training and analysis system of FIG. 2.

DETAILED DESCRIPTION

In some applications, to facilitate analysis of a biological sample, alabel free (i.e., transmitted light) or fluorescent image of thebiological sample may be processed to generate a segmented image. Insuch applications, each pixel of the segmented image corresponds to apixel of the transmitted light or fluorescent image of the biologicalsample. A value of each pixel of the segmented image indicates whether acorresponding pixel of the transmitted light or fluorescent image isassociated with an element of the biological sample that is of interestto a researcher. Such element may include for example, a cell interior,an organelle, a protein, a cell wall, and the like. Because thefluorescent image more clearly distinguishes between elements of thebiological sample, a more accurate segmented image may be produced fromthe fluorescent image.

Further, in other applications, as noted above, a fluorescent image ofthe biological sample may be more readily analyzed to identify cellsthereof that have a particular characteristic (e.g., presence of aparticular protein, whether the cell is alive or dead, etc.) because,for example, the presence of such characteristics causes fluorescencethat is captured in the fluorescent image.

However, as noted above, obtaining a fluorescent image of a biologicalsample may not always be feasible or desirable. As described in detailbelow, a training biological sample may be imaged using transmittedlight and under fluorescence to create a training label free and atraining fluorescent image, respectively. The training fluorescent imagemay be processed to create a segmented image. A machine learning systemmay then be trained to create the segmented image from the traininglabel free image. After training, when the machine learning systemtrained in this manner is presented with a label free image of a furtherbiological sample as an input, the trained machine learning systemproduces a segmented image without the need for a fluorescent image ofthe further biological sample.

Similarly, in some cases, the training fluorescent image is processed todevelop a training characteristic value that indicates the presence of aparticular characteristic in the biological sample represented in thetraining fluorescent image. The training label free image and thetraining characteristic value are used to train the machine learningsystem so that when the machine learning system is presented with alabel free image, the machine learning system generates a characteristicvalue that indicates the presence of the characteristic.

Referring to FIGS. 1 and 2, an imaging system 100 is used by a trainingand analysis system 200 to obtain label free images and fluorescentimages of one or more biological sample(s). In some embodiments, suchimaging system 100 includes an X-Y stage 102, one or more objectivelenses 104, an image capture device 110, one or more mirrors 112, and acontroller 114. For sake of clarity, other components of the imagingdevice 100 may include such as a focus mechanism, one or more filters, aplurality of objective lenses from which the objective lens 104 isselected, and the like are not shown in FIG. 1.

During operation of the imaging system 100, a biological sample carrier116 such as, for example, a microplate, is disposed on the X-Y stage 102either manually or robotically. The sample carrier 116 may include aplurality of wells 118 and a biological sample may be disposed each suchwell 118, wherein each biological sample includes, for example, one ormore cells or portions thereof.

The controller 114 operates the X-Y stage 102 so that a selected well118 or a portion thereof is in the field of view of the imaging device110. The controller 114 then operates the illumination source 106 a toilluminate the selected well 118 with particular wavelengths of lightand actuates the image capture device 110 so that light from theillumination source 106 a that is not absorbed or reflected by thebiological sample disposed in the selected well 118 is transmittedthrough the biological sample, through the objective lens 104, and isrecorded by the image capture device 110 as an image.

If the biological sample is fluorescent (either naturally or by beinglabeled with a fluorescent dye), the controller 114 operates theillumination source 106 b to generate light that causes the biologicalsample to fluoresce, and light emitted by such fluorescence istransmitted through the objective lens 104 and is recorded by the imagecapture device 110 as an image.

One or more mirror(s) 112 (e.g., a semi-transparent, a two-way, and/ordichromic mirror) is/are disposed in the light path between theillumination source 106 b and the sample tray 116 and between the sampletray 116 and the image capture device 110 to direct the light from theillumination source 106 b to the sample tray 116 and from the sampletray 116 to the image capture device 110, respectively.

The controller repeatedly operates the X-Y stage 102, the illuminationsource 106, and the image capture device 110 in this manner until imageshave been captured of all of the wells 118 of the sample tray 116 thatare of interest. Further, the controller 114 may capture several imagesof the same well 118 or portions thereof, when each such image iscaptured under different illumination conditions and/or with differentcombinations of objective lenses 104 and filters (not shown). Bycontrolling the illumination conditions, a first image may be capturedwhen the biological sample is not undergoing fluorescence and a secondimage may be captured when the same biological sample is undergoingfluorescence. Such images may be captured in alignment so that a pixelat each coordinate of the first image and a pixel of the second image atthe same coordinate of the second image correspond to a substantiallyidentical portion of the biological sample.

Although the embodiments described herein refer to the use of amulti-well sample carrier 116, it should be apparent to one havingordinary skill in the art, that a single well carrier or a microscopyslide may be used in connection with these embodiments. Similarly,although these embodiments contemplate the use of X-Y stageautomatically moveable by the controller 114 to capture images ofvarious portions of the sample carrier 116, it should be apparent thatthe X-Y state may be manually moved by an operator (for example, as isdone when using a conventional microscope).

Although FIG. 1 shows the sample tray 116 disposed between theillumination source 106 and the image capture 110, it should be apparentto those of who have skill in the art that the illumination source 106may be disposed between the image capture device 110 and the sample tray116. In such embodiments, light from the illumination source 106reflected by the biological sample or generated by fluorescence of thebiological sample is directed using one or more mirrors (not shown)through the objective lens 104 and captured by the image capture device110.

Further, although the embodiments herein describe the use of alens-based imaging system 100, it should be apparent to one havingordinary skill in the art that images captured using a lens-free imagingsystem may be used in connection with such embodiments.

As described in greater detail below, imaging system 100 may be operatedas described above to provide to the training and analysis system 200label free images of biological samples disposed in the sample tray 116and images of such biological samples under fluorescence. The images ofthe biological samples under fluorescence may be analyzed to identifycells (or portions thereof) of the biological sample in such images thatexhibit a particular characteristic. Such cells identified as beingassociated with the particular characteristic and the transmitted lightimages may be used to train a machine learning system 206, for example,a neural network or another deep learning system. After training, inresponse to an input transmitted light image, the machine learningsystem 206 produces a plurality of characteristic predictions. In someembodiments, each such characteristic prediction is associated with acorresponding pixel in the input image and the value of suchcharacteristic prediction indicates a probability that such pixel isassociated with the particular characteristic. For example, if themachine learning system 206 is trained to identify pixels associatedwith live cells in an input image that includes both live and deadcells, the value of each characteristic prediction generated by themachine learning system 206 indicates the probability that thecorresponding pixel in the input image (and thus the portion of thebiological sample represented by such pixel) is associated with a livecell. In such embodiments, the machine learning system 206 may be usedto, for example, segment an input image based in accordance with thetypes of cell(s) or characteristic(s) of cells present in the image.

In other embodiments, the machine learning system 206 may be trained toanalyze an image to determine if the image represents a particular typeof cell or a cell having a particular characteristic. In suchembodiments, when the machine learning system 206 trained in this manneris provided an image as an input, the machine learning system 206generates an output that is a value that represents a probability thatthe image provided as input includes a cell of the particular type orhaving the particular characteristic. In these embodiments, the machinelearning system 206 may be used to, for example, classify a cellrepresented by an image. An image having a plurality of cellsrepresented therein may be tiled into sub-images of either adjacenttiles or overlapping tiles, wherein each sub-image has dimensionsassociated with a single cell. Such sub-image may then be presented tothe trained machine learning system as an input, and the output of thetrained machine learning system indicates the probability that thesub-image includes a cell of the particular type or having theparticular characteristics.

As shown in FIGS. 1 and 2, in addition to the machine learning system206, the training and analysis system 200 includes an image acquisitionmodule 202 and an image store 204 in which such images may be stored. Insome embodiments, the image acquisition module 202 interfaces with thecontroller 114 to direct the imaging system 100 to capture one or moreimage(s) and provide such captured image(s) to the image acquisitionmodule 202. In some embodiments, the image acquisition module 202instructs the controller 114 to configure the imaging system 100 tocapture a transmitted light image or a fluorescent image using, forexample, particular illuminant(s), objective lens(es), filter(s), andthe like.

The image acquisition module 202 obtains training label free andtraining fluorescent images of training biological samples from theimaging system 100. The training label free and training fluorescentimages are label free and fluorescent images, respectively, that areused to train the machine learning system 206. As described in greaterdetail below, a training module 208 uses the training label free andtraining fluorescent images obtained by the image acquisition module 202from the imaging system 100 and cell characteristics informationassociated with such images developed by an image analysis module 210 totrain the machine learning system 206. After the machine learning system206 is trained, parameters associated with each node of the machinelearning system 206 developed during such training such as activationvalues, scaling factors, kernel weights, and the like are stored in aparameters data store 212. After training, the image acquisition module202 obtains further images of further biological samples, and ameasurement module 214 configures the machine learning system 206 withparameters stored in the parameters data store 212 and operates theconfigured machine learning system 206 to analyze the further images tomeasure characteristics of the further biological samples. In someembodiments, the output generated by the machine learning system 206 maybe processed by an output processing module 216 that thresholds, removesspurious values, and/or otherwise cleanses such output before suchcleansed output is processed by the measurement module 214.

FIG. 3 is a flowchart 250 of the steps undertaken to train the machinelearning system 206. Referring to FIGS. 1-3, at step 252 a user operatesa user computer 218 to specify to the training module a cellcharacteristic to train the machine learning system 206 to identify frominput images. Such characteristics include the presence of, for example,live cells, dead cells, a particular protein or compound, a particularorganelle, and the like. In some embodiments, the user also specifies,using the user computer 218, a type of biological sample (e.g., neuralcells, animal cells, etc.) in which the machine learning system 206 isto be trained to identify the characteristic.

Thereafter, at step 254, a training sample tray 116 that has abiological sample deposited into one or more well(s) 118 thereof iseither manually or robotically loaded onto the X-Y stage 102 of theimaging system 100. Each biological sample(s) deposited into suchwell(s) 118 is stained or otherwise treated with a fluorescent dyeassociated with the characteristic.

After the training sample tray 116 is loaded, the image acquisitionmodule 202, at step 256, directs the imaging system 100 to capture atransmitted light image of each biological sample deposited in thewell(s) 118 of the sample tray 116. In response, the controller 114repeatedly operates the X-Y stage 102 to position a well 118 (or aportion thereof) in the light path between the illumination source 106and the objective lens 104, actuates the illumination source 106 and theimage capture device 110 to capture transmitted light image of the well118 (or portion thereof), and transmits the captured image to imageacquisition module 202. The controller 114 operates the X-Y stage 102,the illumination source, and the image capture device 110 in this manneruntil transmitted light images of all of the wells 118 of the sampletray (or the wells 118 that have a biological sample deposited therein)have been captured and sent to the image acquisition module 202.

Also, at step 256, the image acquisition module 202 receives eachtransmitted light image captured as describe above and stores suchtransmitted light image as a training transmitted light image in theimage data store 204.

At step 258, the image acquisition module 202 instructs the controller114 of the imaging system 100 to cause the biological sample (or dyeaffixed thereto) to fluoresce and capture an image of the biologicalsample during such fluorescence. In response, the controller 114operates the X-Y stage 102 to position the well 118 (or a portionthereof) in the light path between the illumination source 106 and theobjective lens 104. Thereafter, the controller 114 actuates theillumination source 106 to emit light that has one or more particularwavelength(s) selected to cause the fluorescent dye or the biologicalsample to fluoresce, operates the image capture device 110 to capture animage of the biological sample during fluorescence, and transmits thecaptured image to the image acquisition module 202.

In some embodiments, the image acquisition module 202 specifies to thecontroller 114 the wavelength(s) of light the illumination source 106should emit to cause fluorescence of the biological sample and/orfluorescent dye. In these cases, the image acquisition module 202selects these wavelengths in accordance with the characteristic and/orbiological sample type specified by the user at step 252. In otherembodiments, the image acquisition module 202 provides, either at orprior to step 258, the characteristic and/or the biological sample typeto the controller 114 and the controller 114 selects the wavelength(s)of light with which to illuminate the biological sample.

Also, at step 258, the image acquisition module 202 stores each image ofthe biological sample during fluorescence received thereby in the imagedata store 204 as a training fluorescent image.

One or more training fluorescent image is captured for each trainingtransmitted light image. Further, each pixel of the training fluorescentimage corresponds to a pixel of the training transmitted light image,and these pixels represent the intensity of light transmitted through orfluoresced by a substantially identical portion of the biologicalsample.

Although the steps 256 and 258 specify capturing transmitted lightimages of the wells 118 of the sample tray 116 and then capturingfluorescent images of the wells 118, it should be apparent that thecontroller 114 may operate the X-Y stage 102 to position a well 118 inthe light path between the illumination source 106 and the image capturedevice 110, and operate the illumination source 106 and image capturedevice 110 to sequentially capture a transmitted light and fluorescentimages of such well. Thereafter, the controller 114 may operate the X-Ystage 102 to position another portion of the well 118 or another well118 in the light path between the illumination source 106 and the imagecapture device 110 and capture transmitted light and fluorescent imagesthereof repeatedly until all of the wells 118 of the sample trainingtray 116 that have biological samples deposited therein have been imagedin this manner. Each captured image may be transmitted to the imageacquisition module 202 as it is acquired, or all of the captured imagesmay be transmitted in bulk after all such images have been acquired.

Returning to FIG. 3, at step 260 the training module 208 determines ifthere are additional training sample trays to be imaged, for example, byquerying the user via the user computer 218. If there are additionaltraining sample trays, processing proceeds to step 254.

Otherwise, at step 262 the training module uses the training transmittedlight and the training fluorescent images to train the machine learningsystem 206, as described in greater detail below. After training of themachine learning system 206 is complete, at step 264, the parametersassociated with each node of the machine learning system 206 are storedin the parameters data store 212. Such parameters include, for example,activation thresholds, input and output scaling factors or weights,convolution kernels, the architecture of machine learning system 206.For example, if the machine learning system 206 is a neural network, thearchitecture of the nodes or neurons in each layer, interconnectionsbetween node layers, and the like are stored in the parameters datastore 212. As should be apparent to one who has ordinary skill in theart, such neural network may include interconnections of one or moreconvolution layers and one or more neuron layers with one or morepooling layers therebetween. The parameters that are stored aresufficient to configure an untrained machine learning system 206 to atrained state.

FIG. 4 shows a flowchart of the steps the training and analysis system200 undertakes during step 262 of FIG. 3 to train the machine learningsystem 206. Referring to FIG. 4, at step 302, the analysis module 210generates from each fluorescent training image, a corresponding trainingcharacteristic image in accordance with the characteristic specified atstep 252 (FIG. 3). In some embodiments, the training characteristicimage generated includes one pixel for each pixel of the fluorescenttraining image from which such characteristic image is generated. Insuch embodiments, each such pixel indicates a likelihood that acorresponding pixel of the fluorescent training image is associated withthe characteristic.

If the machine learning system 206 is being trained for segmentation,the training characteristic image is segmented using image processingtechniques including boundary (edge) detection, boundary filling, peakdetection, and combinations thereof. In some embodiments, each pixel ofthe segmented training characteristic image corresponds to a pixel ofthe fluorescent training image (and thus a pixel of the transmittedlight image that is associated with the fluorescent training image) and,in some embodiments, has a value of either zero or one. A pixel of thesegmented training characteristic image having a value of one indicatesthat the corresponding pixel of the fluorescent training image isassociated with the selected characteristic. It should be apparent thatother pixel values may be used to identify those pixels of the segmentedtraining characteristic image that are associated with the selectedcharacteristic from other pixels thereof. Thus, the segmented trainingcharacteristic image represents a plurality of training cellcharacteristics (e.g., live cells, dead cells, cells with particularorganelles, particular types of cells, and the like).

At step 304, the training module 208 selects a first subset of thesegmented training characteristic images. In some embodiments, the firstsubset is selected randomly. In other embodiments, the first subset isselected in accordance with a time of capture of the training label freeor fluorescent image associated therewith or with an identifierassociated with one or more sample tray(s) from which the fluorescenttraining image that resulted in the segmented training characteristicimage was developed. It should be apparent to one of ordinary skill inthe art that one or more other criterion/criteria may be used to selectthe first subset. At step 306, the training module 208 selects a firsttraining transmitted light image associated with a first segmentedtraining characteristic image, at step 308, presents the first trainingtransmitted light image as input to the machine learning system 206,and, at step 310, operates the machine learning system 206, to generatea first plurality of predicted training characteristics. The firstplurality of predicted training characteristics includes a predictedtraining characteristic that corresponds to each pixel of the firsttraining transmitted light image and the value of such predictedtraining characteristic represents a probability that the correspondingpixel of the first training transmitted light image is associated withthe characteristic.

At step 312, the training module 208 calculates an error value for eachpredicted training characteristic and a value of a corresponding pixelof the segmented training characteristic image for example using a lossfunction such as, for example, a weighted categorical cross entropyfunction. At step 314, the values of the errors are then used, forexample using backpropagation, to adjust the parameters of the machinelearning system 206, as would be understood by one who has ordinaryskill in the art.

At step 315, the training module 208 determines if there are any imagesof the subset of images selected at step 304 that have not been used asan input to the machine learning system 206. If so, processing proceedsto step 306.

Otherwise, at step 316, the training module 208 evaluates theperformance of the trained machine learning system 206. In particular,the training module 208 selects a second subset of the trainingcharacteristic images and a second subset of the training transmittedlight images corresponding thereto. The training module 208 presents thesecond subset of training transmitted light images to the trainedmachine learning system 206. In response, for each image in the secondsubset, the trained machine learning system 206 generates a set of aplurality of training characteristic predictions, one such plurality foreach second training transmitted light image presented to the trainedmachine learning system 206. A segmented training characteristic imageis developed from each training fluorescent image associated with eachtransmitted light image of the second subset. A set of error values iscalculated between each set of the plurality of training characteristicpredictions and pixels of the segmented training characteristic imageassociated therewith. All of the sets of error values developed in thismanner are combined into an aggregate error metric (e.g., percent ofpixels associated with the characteristic that are identifiedaccurately).

At step 318, the training module 208 compares the aggregate error metricto a predetermined acceptable error, and if the aggregate error metricis greater than the predetermined error, training module 208 proceeds tostep 304 and selects a further subset of images for training asdescribed above. In some embodiments, the training module 208 maydisplay the aggregate error metric on the user computer 218 and querythe operator of such computer whether to undertake further training. Inother embodiments, the training module 218, also at step 318, determineswhether to undertake additional training in accordance with a quantitysamples that have been used for training, a quantity of iterations oftraining that have been undertaken, a rate of improvement in theaggregate error metric between success training passes, an amount oftime undertaken for training, and other such conditions apparent to onewho has ordinary skill in the art. If additional training is warranted,the training module 208 proceeds to step 304.

In some cases, the training module 208, at step 318, may determine thatthe aggregate error metric is greater the predetermined acceptable errorbut that additional training is not warranted (e.g., if the aggregateerror metric is not improving). In such cases, the training module 218may display the aggregate error metric on the user computer 218 with amessage that such aggregate error metric is greater than thepredetermined acceptable error and not undertake additional training.

Referring to FIGS. 2-4, if the training module 208, at step 318 (FIG.4), determines that additional training of the machine learning system206 is not warranted because the aggregate error metric is acceptable orbecause the aggregate error metric is not improving, then step 262 (FIG.3) is completed and the training module proceeds to step 264 (FIG. 3)described above.

The steps 302-318 undertaken by the training and analysis system 200 andshown in FIG. 4 are described above for training the machine learningsystem 206 to segment and/or characterize one or more cells intransmitted light images. Substantially identical steps may beundertaken by the training and analysis system 200 to train the machinelearning system 206 to generate a characteristic value when provided animage as input, wherein the characteristic value indicates a probabilitythat the image includes a cell associated with the characteristicidentified at step 252 of FIG. 3.

In particular, at step 302, instead of developing trainingcharacteristic images from fluorescent images, the analysis module 210develops a characteristic value for each fluorescent image (or sub-imagethereof) that indicates whether the fluorescent image includes a cellhaving the selected characteristic. For example, a characteristic valueof one may indicate a presence of the characteristic and acharacteristic value of zero may indicate absence of the characteristic.It should be apparent that other values may be used instead of one andzero to indicate the presence or absence, respectively, of thecharacteristic. Instead of using segmented training characteristics atsteps 310-314 to train the machine learning system 206, the trainingmodule 208 uses the characteristic value. Further, instead of usingimages having a plurality of cells, the training and analysis system 200may be configured to generate overlapping or adjacent tiled images fromimages obtained at steps 256 and 258 (FIG. 3). Such tiling may beundertaken, for example, by the image acquisition module 202 or theanalysis module 210. In some embodiments, each image may be divided intotiles that have dimensions of, for example, 32 pixels by 32 pixels,wherein each tile duplicates some pixels, e.g. 8 pixels in eachdirection, of adjacent tiles. It should be apparent that tiles of otherdimensions and having different amounts of duplicate pixels may be used.In other embodiments, the dimensions of the tiles may be selected inaccordance with the expected number of pixels a cell or another portionof interest of the biological sample is expected to occupy.

FIG. 5 shows a flowchart 400 of the steps the system 200 undertakes touse the trained machine learning system 206 to measure characteristicsof one or more biological sample(s). Referring to FIG. 5, at step 402, ameasurement module 214 receives from the user computer 218 a selectionof a characteristic to measure in a biological sample, and optionally anidentification of the types of cells that comprise the biologicalsample. In some embodiments, at step 402, the measurement module 214displays on the user computer 218 a list of one or morecharacteristic(s) and/or types of cells for which machine learningparameter(s) have been developed and stored in the parameters data store212. That is, the list indicates the cell characteristic(s) and/or celltype(s) that machine learning system(s) 206 have been trained toidentify in label free images of biological samples. In response, themeasurement module 214 receives from the user computer 218 a selectionof one such characteristic and/or cell type, also at step 402.

At step 404, the measurement module 214 selects the parameters in theparameters data store 212 that resulted from and were stored aftertraining the machine learning system 206 to analyze the selectedcharacteristic in the identified types of cells. Thereafter, themeasurement module 214 configures the machine learning system 206 withthe selected parameters to return the machine learning system 206 to atrained state.

At step 406, a sample tray 116 with the biological sample deposited inthe wells 118 thereof is loaded on the X-Y stage 102. At step 408, theimage acquisition module 202 directs the imaging system 100 to generatea transmitted light image of a well 118, or a portion thereof. Thetransmitted light image is then presented to the trained machinelearning system 206 as an input at step 410 and, in response, thetrained machine learning system 206 produces a plurality of predictedcell characteristics. As described above, a value of each of theplurality of predicted cell characteristics indicates a probability thata corresponding pixel of the transmitted light image is associated withthe characteristic, and thus the portion of the biological samplerepresented in such pixel of the transmitted light image is alsoassociated with the characteristic.

In some embodiments, at step 412, the output processing module 216undertakes additional operations to cleanse the predicted cellcharacteristics. Such cleansing operations involve applying imageprocessing techniques apparent to those who have skill in the artincluding boundary filling, edge detections, and the like. Using suchimage processing techniques, the output processing module 216 may adjustcharacteristic values that are associated with pixels of the transmittedlight that represent edges of cells to make such edges more apparent,remove adjacent characteristic values that represented objects in thetransmitted light image that are too large (or too small) to be a singlecell, and remove characteristic values associated with pixels in thetransmitted light image associated with spurious content. Other types ofcleansing operations apparent to one of ordinary skill in the art may beundertaken at step 412.

In some embodiments, at step 414, the measurement module 214 stores theplurality of predicted cell characteristics generated at step 410 in adata store (not shown), transmits such plurality to another system forfurther analysis, or displays such plurality on the user computer 218.

In some embodiments, the measurement module 214 may generate acharacteristic image from the values of the plurality of predicted cellcharacteristics generated at step 410 at step 416. Each pixel of thecharacteristic image corresponds to a pixel of the transmitted lightimage. In some embodiments, a pixel of the characteristic image is setto a predetermined non-zero value if a corresponding predicted cellcharacteristic exceeds a predetermined threshold and zero otherwise. Inother embodiments, the value of the pixel of the characteristic image isset in accordance to a value of the corresponding predicted cellcharacteristic. For example, the measurement module 214 may set a valueof a pixel of the characteristic image to a particular color associatedwith the probability that the corresponding pixel in the transmittedlight image is associated with the characteristic. In some cases, themeasurement module 214 may automatically develop one or more metric(s)from the characteristic image such as, for example, a count of thenumber of cells or portions thereof represented in the transmitted lightimage have at least a predetermined probability of being associated withthe characteristic. Automatically developing such metric(s) may involveapplying boundary or edge detection, boundary filling, and similaralgorithms to the characteristic image generated at step 416. In someembodiments, the operator selects, for example, from a list, the metricsthe measurement module 214 calculates. In other embodiments, themeasurement module 214 may be preconfigured to calculate particularmetrics. Other ways of selecting the metrics calculated by themeasurement module 214 apparent to those who have skill in the art maybe used. In some embodiments, the characteristic image may be used togenerate a mask that is applied to the transmitted light image toisolate images of cells therein that are associated with thecharacteristic. Additional processing may be undertaken on thecharacteristic image including, for example, local thresholding,competitive growing, boundary growing, and the like.

Returning to FIG. 5, at step 418, the measurement module 214, determinesif all of the wells 118 of the sample tray 116 that have a biologicalsample deposited therein have been processed, and if so, the measurementmodule 214. Otherwise, the training and analysis system 100 proceeds tostep 408 to acquire and analyze additional transmitted light images.

It should be apparent to one of ordinary skill in the art, that thepredicted characteristic values developed at step 410 may simply be asingle characteristic value as described above. In such cases, thecharacteristic value may be thresholded or otherwise processed at step412, and step 416 may not need to be undertaken.

FIG. 6 shows an example of a process flow of training the machinelearning system 206 of the image training and analysis system 200 asdescribed above. Referring to FIGS. 2-6, a training transmitted light(i.e., label free) image 500 of a biological sample is acquired by theimage acquisition module 202 (FIG. 2) at step 256 (FIG. 3) and afluorescent image 502 of the biological sample is acquired by the imageacquisition module 202 at step 258 (FIG. 3). A training characteristicimage 504 is generated by the analysis module 210 (FIG. 2) at step 302(FIG. 4). As described above, the training transmitted light image 500and the training characteristic image 504 are then used by the trainingmodule 208 to train the machine learning system 206, at step 262 (FIG.3).

FIG. 7 shows an example of a process flow of using the trained machinelearning system 206. Referring to FIGS. 2-7, an input transmitted lightimage 506 is acquired by the image acquisition module 202 at step 408(FIG. 5) that is supplied as an input to the trained machine learningsystem 206 at step 410 (FIG. 5). The output from the trained machinelearning system 206 is then used to develop a characteristic image 508by the output processing module 216 (FIG. 2) at step 412 (FIG. 5).

Referring once again to FIG. 2, in some embodiments the machine learningsystem 206 is a convolutional neural network that has an input layerthat accepts as input a 64×64 transmitted light image, and in responsegenerates a 64×64 output image. In some embodiments, the neural network206 is configured using AutoML and NASNet technologies developed byGoogle Inc. of Mountain View, Calif. It should be apparent that otherneural network 206 technologies known to those who have skill in the artmay be used including, for example, a fully convolutional DenseNet,neural networks optimized for machine vision applications, and the like.It should be apparent that the machine learning system 206 may beanother type of machine learning system including a random forest treeand the like.

Although, the training and analysis system 200 described in theforegoing is described as being used to train the machine learningsystem 206 to analyze individual two-dimensional transmitted lightimage, such training and analysis system 200 may be adapted to train themachine learning system 206 to analyze a series of two-dimensionaltransmitted light images of a biological sample taken at different focalpoints that represent a three-dimensional representation of suchbiological sample. In such embodiments, the machine learning system 206is trained using a series of training transmitted light images and aseries of training fluorescent light images, wherein all imagescomprising such series are associated with a substantially identical X-Ylocation of the sample carrier 116, and the corresponding transmittedlight and fluorescent images of the series are associated with adifferent focal point (i.e., different Z location). All of the imagesthat comprise a series are simultaneously provided to the machinelearning system 206 and the machine learning system 206 generates eithera characteristic value that indicates that a cell having a predeterminedcharacteristic is represented by the series transmitted light image, ora set of predicted characteristic values. If the set of predictedcharacteristic values if generated, such set includes one valueassociated with each pixel of the transmitted light images and suchvalue indicates a probability that such pixel is associated with thecharacteristic. Thus, the set of predicted characteristic values may beused to segment in three-dimensions an input series of label-freeimages.

Referring to FIGS. 1 and 2, it should be apparent to one of skill in theart that a first imaging system 100 may be used to supply the label freeand fluorescent images used to train a first untrained machine learningsystem 206 to develop a first trained machine learning system 206, andthe parameters associated with the first trained machine learning system206 may be stored in the parameters data store 212. Thereafter, suchstored parameters may be used to configure a second untrained machinelearning system 206 to develop a second trained machine learning system206 (i.e., replicate the capabilities of the first trained machinelearning system 206 on the second untrained machine learning system206). The second trained machine learning system 206 may be used toanalyze one or more label free image(s) generated using the first or thesecond imaging systems 100. Further, because fluorescent images are notnecessary when operating the first or second trained imaging systems100, it should be apparent that the second imaging system 100 need onlygenerate label free images and, in fact, may not even be equipped togenerate fluorescent images.

It should be apparent to those who have skill in the art that anycombination of hardware and/or software may be used to implement thetraining and analysis system 200 described herein. It will be understoodand appreciated that one or more of the processes, sub-processes, andprocess steps described in connection with FIGS. 1-5 may be performed byhardware, software, or a combination of hardware and software on one ormore electronic or digitally-controlled devices. The software may residein a software memory (not shown) in a suitable electronic processingcomponent or system such as, for example, one or more of the functionalsystems, controllers, devices, components, modules, or sub-modulesschematically depicted in FIGS. 1-5. The software memory may include anordered listing of executable instructions for implementing logicalfunctions (that is, “logic” that may be implemented in digital form suchas digital circuitry or source code, or in analog form such as analogsource such as an analog electrical, sound, or video signal). Theinstructions may be executed within a processing module or controller(e.g., the image acquisition module 202, the machine learning system206, the training module 208, the analysis module 210, and themeasurement module 214, and the processing module 216 of FIG. 2), whichincludes, for example, one or more microprocessors, general purposeprocessors, combinations of processors, digital signal processors(DSPs), field programmable gate arrays (FPGAs), or application-specificintegrated circuits (ASICs). Further, the schematic diagrams describe alogical division of functions having physical (hardware and/or software)implementations that are not limited by architecture or the physicallayout of the functions. The example systems described in thisapplication may be implemented in a variety of configurations andoperate as hardware/software components in a single hardware/softwareunit, or in separate hardware/software units.

It should be apparent to one who has skill in the art that one or morecomponents of the image training and analysis system 200 may operate ina cloud environment, for example, using distributed computing andstorage systems connected over a local or wide area network (e.g., theInternet or a private network). For example, one or more of the imageacquisition module 202, the machine learning system 206, the trainingmodule 208, the output processing module 216, and the measurement module214 may operate on one or more computer(s) remote from the user computer218, the imaging system 100, and/or one another. In one suchconfiguration, for example, the image acquisition module 202 maycommunicate with the controller 114 over the local or wide area networkto acquire images from the imaging system 100 and store images acquiredin this manner in a cloud-based image store 204. The training module 208operating locally or in the cloud may access such images to train themachine learning system 206 and store the parameters in the parametersdata store 212, which may be also be a cloud-based storage system. Themeasurement module 214 (operating on a local server or in the cloudenvironment) may access the parameters stored in the parameters datastore 212 to train an untrained machine learning system 206 (operatingon a local computer or in the cloud environment). Similarly, operationof the machine learning system 206 may be undertaken to analyze imagesacquired using a local or remote imaging system 100.

The executable instructions may be implemented as a computer programproduct having instructions stored therein which, when executed by aprocessing module of an electronic system, direct the electronic systemto carry out the instructions. The computer program product may beselectively embodied in any non-transitory computer-readable storagemedium for use by or in connection with an instruction execution system,apparatus, or device, such as an electronic computer-based system,processor-containing system, or other system that may selectively fetchthe instructions from the instruction execution system, apparatus, ordevice and execute the instructions. In the context of this document,computer-readable storage medium is any non-transitory means that maystore the program for use by or in connection with the instructionexecution system, apparatus, or device. The non-transitorycomputer-readable storage medium may selectively be, for example, anelectronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, or device. A non-exhaustive list ofmore specific examples of non-transitory computer readable mediainclude: an electrical connection having one or more wires (electronic);a portable computer diskette (magnetic); a random access, i.e.,volatile, memory (electronic); a read-only memory (electronic); anerasable programmable read only memory such as, for example, Flashmemory (electronic); a compact disc memory such as, for example, CD-ROM,CD-R, CD-RW (optical); and digital versatile disc memory, i.e., DVD(optical).

It will also be understood that receiving and transmitting of signals ordata as used in this document means that two or more systems, devices,components, modules, or sub-modules are capable of communicating witheach other via signals that travel over some type of signal path. Thesignals may be communication, power, data, or energy signals, which maycommunicate information, power, or energy from a first system, device,component, module, or sub-module to a second system, device, component,module, or sub-module along a signal path between the first and secondsystem, device, component, module, or sub-module. The signal paths mayinclude physical, electrical, magnetic, electromagnetic,electrochemical, optical, wired, or wireless connections. The signalpaths may also include additional systems, devices, components, modules,or sub-modules between the first and second system, device, component,module, or sub-module.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

The use of the terms “a” and “an” and “the” and similar references inthe context of describing the invention (especially in the context ofthe following claims) are to be construed to cover both the singular andthe plural, unless otherwise indicated herein or clearly contradicted bycontext. Recitation of ranges of values herein are merely intended toserve as a shorthand method of referring individually to each separatevalue falling within the range, unless otherwise indicated herein, andeach separate value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g., “such as”) provided herein, isintended merely to better illuminate the disclosure and does not pose alimitation on the scope of the disclosure unless otherwise claimed. Nolanguage in the specification should be construed as indicating anynon-claimed element as essential to the practice of the disclosure.

Numerous modifications to the present disclosure will be apparent tothose skilled in the art in view of the foregoing description. It shouldbe understood that the illustrated embodiments are exemplary only andshould not be taken as limiting the scope of the disclosure.

We claim:
 1. A method of analyzing a biological sample using an imagingsystem comprising the step of: causing a programmable device toundertake the steps of: instructing the imaging system to obtain a labelfree image of a biological sample, and in response receiving the labelfree image; instructing the imaging system to cause the biologicalsample to fluoresce and obtain a fluorescent image of the biologicalsample undergoing fluorescence, and in response receiving thefluorescent image; analyzing the fluorescent image to generate asegmented image, wherein a value of each pixel of the segmented imageindicates whether a corresponding pixel of the label free image isassociated with a particular characteristic; and training a machinelearning system using the label free image and the segmented image todevelop a trained machine learning system such that when the trainedmachine learning system is operated with the label free image as aninput, the trained machine learning system generates a plurality ofpredicted cell characteristics, wherein each predicted cellcharacteristic of the plurality of predicted cell characteristicscorresponds to a pixel of the label free image and a value of thepredicted cell characteristic indicates a probability that the pixel isassociated with the particular characteristic.
 2. The method of claim 1,wherein the programmable device iteratively undertakes the step oftraining the machine learning system until a difference between each ofthe plurality of predicted cell characteristics and a correspondingpixel of the segmented image is within a predetermined amount.
 3. Themethod of claim 1, wherein the biological sample comprises a firstbiological sample, the label free image comprises a first label freeimage, and the plurality of predicted cell characteristics comprises afirst plurality of predicted cell characteristics and further causingthe programmable device to undertake the steps of receiving a secondlabel free image of a second biological sample, operating the trainedmachine learning system with the second label free image as an input,and in response receiving a second plurality of cell characteristics,wherein each one of the second plurality of predicted cellcharacteristics indicates whether a corresponding pixel of the secondlabel free image is associated with the particular characteristic. 4.The method of claim 3, wherein causing the programmable device toundertake the step of receiving the second plurality of predicted cellcharacteristics comprises the step of receiving a cell characteristicfor each pixel of the second label free image.
 5. The method of claim 3,wherein the programmable device undertakes the further step ofprocessing the second plurality of predicted cell characteristics todevelop metrics associated with the second label free image.
 6. Themethod of claim 5, wherein the metrics comprise a count of one of anumber of live cells, dead cells, particular organelles, and particulartypes of cells represented in the second label free image.
 7. The methodof claim 1, wherein the trained machine learning system comprises afirst trained machine learning system and the programmable devicecomprises a first programmable device, and further including the step ofcausing the first programmable device to undertake the further step ofstoring parameters associated with the first trained machine learningsystem, and causing a second programmable device to undertake the stepof configuring an untrained machine learning system with the storedparameters to develop a second trained machine learning system.
 8. Themethod of claim 1, wherein the biological sample comprises a firstbiological sample, the label free image comprises a first label freeimage, the programmable device comprises a first programmable device,and the plurality of predicted cell characteristics comprises a firstplurality of predicted cell characteristics, and the method includescausing a second programmable device remote from the first programmabledevice to undertake the steps of receiving a second label free image ofa second biological sample, and causing the first programmable device toundertake the step of operating the trained machine learning system withthe second label free image as an input and in response receiving asecond plurality of predicted cell characteristics, wherein each one ofthe second plurality of predicted cell characteristics indicates if acorresponding pixel of the second label free image is associated withthe particular characteristic.
 9. The method of claim 1, wherein causingthe programmable device to train the machine learning system includesundertaking the step of training the machine learning system to analyzea particular type of biological sample for the particularcharacteristic.
 10. The method of claim 1, wherein the label free imageis one of a series of label free images that represent athree-dimensional representation of the biological sample, and furtherincluding causing the non-transitory programmable device to train themachine learning system includes simultaneously providing all of theseries of label free images to the machine learning system as an input.11. A system for analyzing a biological sample using an imaging systemcomprising: a machine learning system operating on at least oneprogrammable device; an image acquisition module operating on the atleast one programmable device that instructs the imaging system togenerate a label free image of a biological sample, and in responsereceives the label free image, and instructs the imaging system to causethe biological sample to fluoresce and obtain a fluorescent image of thebiological sample undergoing fluorescence, and in response receives thefluorescent image; an analysis module operating on the at least oneprogrammable device that analyzes the fluorescent image to generate asegmented image, wherein a value of each pixel of the segmented imageindicates whether a corresponding pixel of the label free image isassociated with a particular characteristic; and a training moduleoperating on the at least one programmable device that trains themachine learning system using the label free image and the segmentedimage to develop a trained machine learning system such that when thetrained machine learning system is operated with the label free image asan input, the trained machine learning system generates a plurality ofpredicted cell characteristics, each predicted cell characteristic ofthe plurality of cell characteristics corresponds to a pixel of thelabel free image, and a value of the predicted cell characteristicindicates a probability that the pixel is associated with the particularcharacteristic.
 12. The system of claim 11, wherein the training moduleiteratively undertakes the step of training the machine learning systemuntil a difference between each of the plurality of predicted cellcharacteristics and a corresponding pixel of the segmented image iswithin a predetermined amount.
 13. The system of claim 11, wherein thebiological sample comprises a first biological sample, the label freeimage comprises a first label free image, and the plurality of predictedcell characteristics comprises a first plurality of predicted cellcharacteristics and the image acquisition module receives a second labelfree image of a second biological sample and a measurement moduleoperating on the at least one programmable device operates the trainedmachine learning system with the second label free image as an input,and in response receives a second plurality of predicted cellcharacteristics, wherein each one of the second plurality of cellcharacteristics indicates whether a corresponding pixel of the secondlabel free image is associated with the particular characteristic. 14.The system of claim 13, wherein the measurement module operates on afirst one of the at least one programmable device that is remote from atleast another one of the non-transitory programmable device on which themachine learning system, the image acquisition module, the analysismodule, and the training module operate.
 15. The system of claim 13,wherein the second plurality of predicted cell characteristics comprisesa predicted cell characteristic associated with each pixel of the secondlabel free.
 16. The system of claim 13, wherein the measurement moduleprocesses the second plurality of predicted cell characteristics todevelop metrics associated with the second label free image.
 17. Thesystem of claim 11, wherein the biological sample comprises a firstbiological sample, the label free image comprises a first label freeimage, the plurality of predicted cell characteristics comprises a firstplurality of predicted cell characteristics, the at least oneprogrammable device includes a first programmable device and a secondprogrammable device, and the trained machine learning system includes afirst trained machine learning system operating on the firstprogrammable device, and further including a further image acquisitionmodule and a measurement module operating on the second programmabledevice, wherein the training module stores parameters associated withthe first trained machine learning system, the measurement moduleconfigures an untrained machine learning system with the storedparameters to develop a second trained machine learning system, whereinthe further image acquisition module receives a second label free imageof a second biological sample and the measurement module operates thesecond trained machine learning system with the second label free imageas an input and, in response receives a second plurality of predictedcell characteristics associated with the second label free image. 18.The system of claim 11, wherein the machine learning system comprises aconvolutional neural network.
 19. The system of claim 11, wherein thetraining module trains the machine learning system to analyze aparticular type of biological sample for the particular characteristic.20. The system of claim 11, wherein the label free image is one of aseries of label free images that represent a three-dimensionalrepresentation of the biological sample, and the training module trainsthe machine learning system by simultaneously providing all of theseries of label free images to the machine learning system as an input.